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ABSTRACT 

The purposes of this study were to assess the 
usefulness of a variety of readability formulas in predicting the 
relative difficulty of passages, and to explore the contribution of 
pupils' background to text difficulty. Subjects were 285 special 
education students in gradesl-9, 117 of whom were based in rural and 
suburban Minnesota (MN) and lfT8 of whom lived in New York City (NYC). 
Each student was tested twice' on three passages of a Passage Reading 
Test (PRT); Analyses of variance applied to the data revealed an 
interaction between passages and student site (MN^vs NYC), suggesting 
that text difficulty was affected by students' background. 
Additionally, six readability formulas were applied to the three PRT 
passages. There was no agreement' between passage difficulty rankings 
derived from the formulas and those based on students' actual 
performance. Implications for; creating readability formulas are 
discussed. (Authcjx-) 
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Abstract 

The purposes of this study were to assess the usefulness of a 
variety of readability formulas in predicting the relative difficulty 
of passages, and to explore the contribution of pupils' background to 
text difficulty. Subjects were 285 special education students in 
grades 1-9, 117 of whom were based in rural and suburban Minnesota 
(MN) and 168 of whom lived in New York City. (NYC). Each student was 
tested twice on three passages of a Passage Reading Test (PRT). 
Analyses of variance applied to the data revealed an interaction 
between passages and student site (MN vs NYC),, suggesting that text 
difficulty was affected by students' background'. Additionally, six 
readability formulas were applied to the three PRT passages. There 
was no agreement between passage" difficulty rankings derived from the 
formulas and those based on students' actual performance. 
Implications for creating readability formulas are discussed. 
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The Nature of Inaccuracy Among Readability Formulas 
Formulas for measuring the readability of a texts have been 
devel oped and empl oyed w1 dely for at 1 east four decades (Kemper, 
1983). Such formulas may enjoy broad use due to their simplicity and 
Vho ease with which educators can be taught to apply them, as well as 
because of the appeal of such a seemingly scientific approach to 
describing passage difficulty. 

Teachers rely on readability formulas for determining appropriate 
instructiona*! reading material , and evidence (Stevens & Roserrshine, 
1981) suggests that the activity of matching reading material 
correctly with students 1 skills is important to student achievement. 
Nevertheless, the methods by which readability formulas are derived 
and the type of variables they include to predict text difficulty have 
been criticized (Fitzgerald, 1980; Fuchs, Fuchs, & Oeno, 1982; Kemper, 
1983).. The objective of . the present study was to address two 
frequently cited criticisms: (a) the poor accuracy of readability 
formulas, and (b) the serious omission from currently available 

formulas of variables that describe students 1 background or 

x 

familiarity with text. Specifically, the study had two purposes: to 
assess the usefulness of a variety of readabil ity formulas in 
predicting the relative difficulty of passages, using students 1 actual 
reading scores to determine the criterion difficulty of the passages, 
and to explore the contribution of pup.ls' backgrounds to text 
difficulty. 

Method 

Subjects •> 

Subjects were 285 students distributed across grades 1-9. Of 
this group, 117 (92 males, 23 females, 2 unreported) were in special 
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education programs 1n rural and suburban Minnesota public schools. 
The average grade level of these students was 3.76 (SD •» 1.39). 
Another 16# pupils (129 males, 39 females) were in New York City 
PubMc School special education programs. The average grade level of 
these children was 5.62 (SD = 1.66). Statistical tests revealed that 
the Minnesota and New York students Were similar with respect to sex, 
X 2 (D= .60, ns, but there was a statistically significant difference 
between the grade levels of the two groups, t(283) = 10.00, £ < .001. 
Measures 

A passage reading test (PRT; Fuchs, Deno, & Mirkin, in press), 
comprised of three reading passages from a third grade book of the 
Ginn 720 series (1976), was employed. Two passages were sampled 
randomly from the text and one was chosen to represent the difficulty 
of, the last 25% of the book. (See Fuchs et al., 1982, for selection 
procedure.) The test requires students to read aloud from each 
passage for one minute, while the examiner marks omissions, 
substitutions, insertions, and mispronunciations. Student performance 
was reported in terms of the number of correct words read on each 
passage. Test-retest reliability ranged from .93 to .96 (Fuchs, Deno, 
& Marston, in press) . Concurrent validity with respect to the 
Woodcock Reading Mastery Tests, Word Identification and Passage 
Comprehension Tests, ranged between .89 and .92 (Fuchs, 1981). 
Internal consistency reliability (Cronbach's alpha) for the three 
passage test was .79 (Fuchs, Deno, & Mirkin,' in. press). 
Readability Formulas 

Six readability formulas ^were employed. Table 1 lists the 
developers of the formulas as well as the predictor variables employed 
in each. 7 
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Insert Table 1 about here 



Procedure 

Each readability formula was applied to the passages of the PRT. 
Then, each student was tested individually on the three passages of 
the PRT, in standard fashion (see Mirkin, Deno, Fuchs, Wesson, Tindal, 
Marston, & Kuehnle, 1981), The PRT was administered twice to each 
student, with seven months intervening between the administrations. 
Data Analysis 

The three passages of the PRT were assigned rank orders of 
difficulty according to (a) students 1 actual mean performances on each 
administration of the PRT, ano (b) scores derived from the application 
of the six readability formulas. Then, a one between factor (site: 
Minnesota vs. New York City), repeated measures (passages 1 vs. 2 vs. 
3) analysis of variance was run on scores from each administration of 
the PRT. 

Results 

Table 2 presents the formula readabil ity scores, the actual 
reading scores, and the difficulty rartk orderings of the three PRT 
passages. On both administrations of the PRT, students' average o 
scores on the passages were ranked in the same order. However, there 
was consistent disagreement , between these rankings and those Ifrom the 
application of the formulas. Among the six readability score rank 
orderings, the Spache formula and the Fog Index agreed with each other 
and the Coleman and GllUland formulas agreed with each other. 

. •• / 



Insert Table 2 about here 

Employing the rank ordering derived from' students' 1 actual 
performance as the criterion difficulty rankings, the percentages of 
Increase 1n difficulty between successively ordered PRT passages were 

computed (see Table 3). As measured by students' actual scores across 

\« 

the two PRT administrations, difficulty Increased an average 6.5* and 

\ 

an average 12.0*, respectively, between passages 1 and 2 an d^ between 
passages 2 and 3. The readability score estimates > of difficulty 
Increases between successive passages were variable. The average 
absolute value of the discrepancy between the , difficulty Increases 
Indicated by students' actual scores and by the readability scores was 

9.08* (SD = 6.46) across the six formulas and across the two 

I 

comparisons (passages 1 vs. 2 and 2 vs. 3). | 

Insert Table 3 about here 



Means and standard deviations of students' scores on each passage 
for each administration of the PRT are reported by site 1n Table 4. A 
one between (site), repeated measures (passages) analysis of variance, 
applied to the first PRT administration, revealed significant main 
effects for both site, F(l,282)= 9.81, p < .01, and passages, F(2,564) 
= 76.20, £ < .001, as well as an F value that approached significance 
for the site X passage Interaction, £(2,564) = 2i71, £ = .067. 
Applied to the second administration of the PRT, the analysis of 



variance revealed significant effects for passages, F(2,538) ■ 55.75, 
£ < .001, and for the site X passage Interaction, F(2,538) » 16.62, £ 
< .001. There was no significant effect for the site factor, F(l,269) 
* 2.89, ns. 



Insert Table 4 about here 



Follow-up, orthogonal correlated t tests on the passage factor 
revealed that there was a statistically significant difference between 
performance on passages 1 and 2 vs. passage 3, t,(293) = 10.30, £ < 



.001 and t(279 
administrations 
statistically si 



) * 8.76, £ < .001, for the first and second 
of the PRT, respectively. Additionally, there was a 
gnlflcant difference between performance on passage 1 



vs. passage 2, t(292) = 4.56, £ < .001 and t(279) = 5.10, £ < .001, 
for the first and second PRT administrations, respectively. 

To clarify the site X passage interaction on the two PRT 
administrations, the percentages of increase 1n difficulty between 
successive PRT passages for the two sites were calculated (see Table 
5). Figures 1 and 2 display the interactions. As suggested in Table 
5 and Figures 1 and 2, the difficulty between passages 1 and 2 was 
similar for the two groups. However, the difference in difficult;/ 
betwein passages 2 and 3 was much greater for the Minnesota pupils 
relative to the New York City group. 



Insert Table 5 and Figures 1 and 2 about here 
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D iscussion 

The first purpose of this study was to assess the usefulness of 
readability formulas 1n predicting the relative difficulty of 
passages, using students 1 actual reading scores to determine the 
criterion difficulty of the passages. Findings suggest that the 
readability formulas were Inaccurate. None of the rank orderings of 
passage difficulty based on the formulas agreed with the one suggested 

by students 1 actual average performances on the passages. The 

i 

soundness of the students' actual performance to derive a reliable 
criterion ranking of passage difficulty 1s Indicated 1n two ways: 
first, by the fact that students' average scores on. the passages were 
ranked 1n the, same order over the two PRT administrations; second, 
because orthogonal t tests revealed significant differences 1n 

students 1 performances among the three passages for both PRT 

i 

administrations. In addition to predicting poorly the criterion 
difficulty rank order of the passages, the readability formulas failed 
to agree meaningfully with each other. Although two pairs of formulas 
produced the same rank .orderlngs, Inspection of Table 1 Indicates that 
neither the Spache-Gunning pair nor the Coleman-Gilliland pair 
includes the same predictor variables. Finally, the readability 
formulas Inaccurately estimated the percentages of Increase 1n 
difficulty between successively more difficult passages. On average, 
the formulas missed the criterion Indices of difficulty Increases by 
over 9%. 

These findings raise serious questions about the accuracy and 
usefulness of readability formulas In predicting passage difficulty, 
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and may at least partially be explained by the methods by which such 
formulas are developed. Typically, readability formulas have been 
derived and refined by their success 1n predicting difficulty 
estimates of basal texts or older versions of the McCall-Crabbs (1925, 
1950, 1961; cited in Fitzgerald, 1980) Standard Test Lessons 1n 
Reading (Klare, 1974-1975). Although it has been assumed that basal 
/ texts and the Test Lessons were developed adequately for use as the 
criterion of accuracy for readability formulas (Klare, 1974-1975), 
there actually 1s little evidence to support either the correctness of 
text readability designations (Bradley & Ames, 1977; Britton & 
Lumpkin, 1977; Fitzgerald, 1980; Fuchs et al., 1982) or the integrity 
of the Test Lessons' standardization (see Fitzgerald, 1980). 

Additionally, readability formulas have been criticized (Kemper, 
1983; Pearson, 1974-1975) because they rely on surface characteristics 
such as word length, word frequency, . sentence length, and sentence 
complexity (see Table 1). Formulas typically fail to. account for 
students 1 familiarity with text (Fuchs et al., 1982), and therefore 
ignore the contributions of the reader in terms of general background 
knowledge or expertise. Such a reliance on the mechanics of text, 
rather than the content of a passage or the skills of a reader, may 
explain at least partially the failure of the readability formulas to 
predict students 1 actual performance on the passages. 

In investigating the effect of students 1 Dackground ,on 
readability, pupils' school site was designated the critical factor: 
Minnesota children were conceptualized as having an essentially 
rural/suburban experience whereas New York City youngsters were viewed 

w 9 
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as having big city expertise. Findings of the study suggest that 
students' site, or background, did account for the relative difficulty, 
of the passages. The difference in students' scopes, between passages 
1 and 2 were similar; however, passage 3 relative to passage 2 was 
mi^h harder for the Minnesota pupils than for the New York City 
pupils 1 . Inspection of the content of passage 3 revealed that the text 
1 describes gang activity and a city bus ride, essentially urban 
experiences. This suggests that the interaction between site and 
passage, demonstrated on the second PRT administration and approached 
on the first PRT testing, - may be explained by the relation between 
reader background and text content. ,"■ 

Consequently, findings of this study indicate that developers :of 
readability formulas might consider (a) employing students 1 actual 
reading scores, rather than questionable difficulty designations of 
passages, as thejr criterion variables, and (b) developing and 
incorporating predictor Variables that address a reader's background 
or expertise. A recent attempt at developing a readability formula 
(Kemper, 1983) included' a measure of the inference load of a text, 
which reflects the difficulty readers have in inferring the causal 
connections necessary to recover the event c .chains underlying texts. 
This represents both a move beyond the surface cl^ra^t eristics of text 
and a wor^h^hile attempt at assessing the comprehens%bij;ity of a 
passage. Nevertheless, it still fails to incorporate aspects of 
student background into readabil ity formulas, a variable that may be 



critical in determining the^ 
students.-- 



difficulty of reading material for 
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Table 1 r ■ 

Predictor Variables Included in Readability Formulas 



• i . Predictor Variables I 

' 1 

/ Diffi- Preposi- 1- 3- . j' 

Sentence cult tional Syllable Syllable. |. 

eveloper Length Words Phrases Words Words Pronouns (Syllables , Sentences 



pache (1953) 

orge (1948) 

arr- Jenkins-Patterson 
(f951) 

uniiing (1952; The Fog 
Index) 

oleman (1971) 
illiland 0974) 



\ 



16 



J 



17 



o 



12 



\ 
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Table 2 . 

Formula Readability Scores, Actual Reading Scores, and 
Difficulty Rank Orders* of Three Ginn 720, Grade 3 
Passages of the PRT 



s 






Passage 




Formula Readability' 5 


1 


2 


3 . 


Spache (1953) 


4.92(3) 


x 4.36(2) 


3.66(1 ) 


Lorge (1948) 


5.38(1) 


5.61(3) 


5.42(2) 


Farr- Jenkins-Patterson (1951)* 


67.17(2) 


65.85(1) 


71.74(3) 


Gunning (1952; The Fog Index) 


9.02(3) 


7.65(2) 


5.14(1) 


Ctfleman (1971 )* 


59.47(3) 


52.62(1) 


58.79(2) 


Gill 11 and (1974) 


7.25(3) 


- 6.25(1) 


7.00(2) 

i. 


Actual Reading 0 








Timel 


44.13(1) 


41.44(2) _ 


'- 35.34(3) 


Time 2 


61 .07(1) 


56.83(2), ... 


, 51.57(3) 



a Rank order assignments are indicated in parentheses (l=easiest). 

^Readability scores are in terms of grade le,ye1 except where indicated 
with an asterisk(*) , raw scores are reported. 

C N=280. 
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Table 3 



Percentages of Increase in Difficulty Between Successive PRT 
Passages as Indicated by Readability Scores and Actual Reading Scores 




Percentage Increase Between Passages 



Formula' Readability ' * 1 and 2 2 and 3 



Spache (1953). . ' 11 .4 16.1 

Lorge (1948) -4.3 \ 3.4 

Farr- Jenkins-Patterson (1951) 2.0 \ -8.9 

Gunning (1952; The Fog Index) 15.2 \ 32„8 

Coleman (1971) • 11.5 \ -11.7 

Gilliland (1974) - 13.8 -1.2 - 

Actual Reading . ' 

Time 1 6.1 14.7 

Time 2 6.9 9.3 
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Table 4 

Students' Scores on PRT Passages for Each Testing by Site 





Site 

Minnesota 


New 


York 


M 


SD 


M 


SD 


Time l a 










Passage 1 


38.45 


26.02 


49.58 


41 .44 


Passage 2 


35.66 


24.95 


46.99 


41.38 


Passage 3 


28.09 


19.44 


42.28 


40.01 


Time 2 b 










Passage 1 


57.75 


31.99 


64.13 . 


48.02 


Passage 2 


54.79 


31.83 


58.71 


43.78 


Passage 3 


44.26 


26.56 


58.31 


50.79 



a N=284~. 
b N=271 . 




Table 5 • 

Percentages of Increase in Difficulty Between Successive 

Passages by Site 













1 


Site 








0 


Minnesota 


New York 


Time 1 












Passages 


1 


vs. 2 




7.3 


; . 5.2 ' 


Passages 


2 


vs. 3 




21 .2 


10.0 


Time 2 












Passages 


1 


vs. 2 




5.T 


8.5 


Passages 


2 


vs. 3 




19.2 


.. .7 




r - 



T ! r 

1 2 3 

PAS SAGE 



Figure 2. Scores on passages 1, 2, and 3 for New York ( ) and Minnesota 

(---) students on the second PRT administration. 
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